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Abstract —In this paper, we study the stability problem of a 
stochastic, nonlinear, discrete-time system. We introduce a linear 
transfer operator-based Lyapunov measure as a new tool for 
stability verification of stochastic systems. Weaker set-theoretic 
notion of almost everywhere stochastic stability is introduced 
and verified, using Lyapunov measure-based stochastic stability 
theorems. Furthermore, connection between Lyapunov functions, 
a popular tool for stochastic stability verification, and Lyapunov 
measures is established. Using the duality property between the 
linear transfer Perron-Frobenius and Koopman operators, we 
show the Lyapunov measure and Lyapunov function used for 
the verification of stochastic stability are dual to each other. 
Set-oriented numerical methods are proposed for the finite 
dimensional approximation of the Perron-Frobenius operator; 
hence, Lyapunov measure is proposed. Stability results in finite 
dimensional approximation space are also presented. Finite 
dimensional approximation is shown to introduce further weaker 
notion of stability referred to as coarse stochastic stability. The 
results in this paper extend our earlier work on the use of 
Lyapunov measures for almost everywhere stability verification of 
deterministic dynamical systems (“Lyapunov Measure for Almost 
Everywhere Stability”, IEEE Trans, on Automatic Control, Vol. 53, 
No. 1, Feb. 2008). 

Index Terms — Stochastic stability, almost everywhere, compu¬ 
tational methods. 


I. Introduction 

Stability analysis and control of stochastic systems are a 
problem of theoretical and applied interests. For stochastic 
systems, there are various notions of stabilities. Among the 
most popular notions of stabilities are almost sure and moment 
stability 0 ^ 0 - Almost sure notion of stability implies 
stability of sample-path trajectories of stochastic systems and 
is a weaker notion of stability compared to the moment 
stability definition. Moment stability definition deals with the 
steady state probability density function of a stochastic system, 
in particular, the integrability of it with respect to various 
powers of state. There is extensive literature on stochastic 
stability and stabilization in control and dynamical system 
literature. Some of the classic results on stability analysis and 
control of stochastic dynamical system are found in 0-0- 
Mao Q presents a systematic summary of results on various 
stochastic stability definitions and Lyapunov function-based 
verification techniques for a stochastic system. The topic of 
stochastic stability of switched system and the Markov jump 
system has also attracted lots of attention with applications in 
network controlled dynamical systems 0-lID- The results in 

Financial support from the National Science Foundation grant ECCS- 
1150405 and CNS-1329915 is gratefully acknowledged. U. Vaidya is with the 
Department of Electrical and Computer Engineering, Iowa State University, 
Ames, lA, 50011 USA e-mail: ugvaidya@iastate.edu. 


these papers address not only the stochastic stability problem, 
but also robust control synthesis and fundamental limitations 
issues that arise in stabilization and estimation of dynamical 
systems in the presence of stochasticity in a feedback loop. 
More generally, stochastic stability and stabilization problems 
for a nonlinear system with a multiplicative noise process are 
addressed in GH-iiTg. 

As in a deterministic system, most of the existing methods 
for stochastic stability verification are based directly or indi¬ 
rectly on the Lyapunov function. The Lyapunov function is 
used for both almost sure and moment stability verification. 
The Lyapunov function and Lyapunov function-based methods 
are also applied for stochastic stabilization While 

there is extensive literature on the numerical procedure for the 
construction of a Lyapunov function for stability verification 
and stabilization of a deterministic dynamical system m- 
1^, the literature on numerical methods for stochastic stabil¬ 
ity verification is very scant | [3T| , p2) . The contributions of 
this paper are two-fold. We provide a novel operator theoretical 
framework for stability verification of a stochastic dynamical 
system. We also show the proposed operator theoretic frame¬ 
work is amicable to computations thereby providing systematic 
numerical procedures for stochastic stability verification. 

The operator theoretic framework that we introduce in this 
paper is used to verify a weaker set-theoretic notion of almost 
everywhere stability of a stochastic system. The notion of 
almost everywhere stability was introduced for the first time 
in the work of Rantzer for continuous time deterministic 
dynamical systems pT| . This notion of stability was later 
extended to continuous time stochastic systems in p^ . The 
notion of a.e. stability essentially implies the set of points in 
the state space starting from which system trajectories are not 
attracted to the attractor set is a measure zero set. The almost 
everywhere notion of stability was also used for the design 
of stabilizing feedback controller and for solving verification 
problems for a nonlinear system p2) . The input-output version 
of almost everywhere stability was developed in p5) . In p^ , 
linear transfer operator-based framework was introduced to 
verify this weaker set-theoretic notion of a.e. stability for 
discrete-time dynamical system. Transfer Perron-Frobenius 
operator-based Lyapunov measure was introduced as a new 
tool to verify a.e stability of a nonlinear system. Duality 
between the Lyapunov function and Lyapunov measure was 
established using the duality in the transfer Perron-Frobenius 
and Koopman operators. Application of the Lyapunov measure 
for the design of stabilizing and optimal feedback controller 
was proposed in pTj , p8) . Operator theoretical framework 
involving spectral analysis of Koopman operator was proposed 
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in p9| for stability analysis of deterministic nonlinear systems. 
The results in this paper can be viewed as a natural extension 
of the results from towards a.e. stability verification of 
discrete time stochastic dynamical system. 

Associated with the stochastic dynamical system are two 
linear transfer operators, the Perron-Frobenius (P-F) and 
Koopman operators. These operators are used to study the 
evolution of ensembles of points in the form of measures or 
density supported on state space. These operators are dual to 
each other and are used in the dynamical system literature 
for the analysis of deterministic and stochastic dynamical 
systems |4§-||43. One of the main contributions of this 
paper is the introduction of a linear transfer P-F operator 
based Lyapunov measure for a.e. stochastic stability veri¬ 
fication of stochastic dynamical system. We introduce a.e. 
almost sure notion of stability and provide Lyapunov measure- 
based stability theorems to verify this stability. The Lyapunov 
function for stochastic stability verification is shown intimately 
connected with Koopman operator formalism. In particular, 
analytical formulas for the computation of Lyapunov measure 
and the Lyapunov function are obtained in terms of resolvent 
of the Perron-Frobenius and Koopman operator respectively. 
By exploiting the duality relationship between Koopman and 
Perron-Frobenius operators, we show the Lyapunov function 
and Lyapunov measure are dual to each other in a stochastic 
setting as well. The results presented in this paper are extended 
version of results appeared in pSj , | |49) . 

While results exist for the application of operator theoretical 
methods for the stability verification of a stochastic system in 
more general Markov chain settings | [50) , the main motivation 
of this work is to provide computational methods for the 
construction of stability certificate in the form of a Lyapunov 
measure. Towards this goal, set-oriented methods are used for 
the finite dimensional approximation of the linear transfer P-F 
operator and for computation of the Lyapunov measure 0, 
| [52| . The finite dimension approximation introduces a further 
weaker notion of a.e. stability by allowing stable dynamics 
in the complement of the attractor set, but their domain of 
attraction is smaller than the size of discretization cells used in 
the finite dimensional approximation. This notion of stability is 
referred to as coarse stochastic stability. The finite dimensional 
approximation of the P-F operator arises as a Markov matrix 
and provides for various alternate formulas for the computation 
of the Lyapunov measure. 

Organization of this paper is as follows. In section 
we discuss the preliminaries of the transfer operators and 
introduce various set-theoretic stochastic stability definitions. 
The main results of this paper on Lyapunov measure-based, 
stochastic stability theorems are proven in section The 
formula for obtaining the Lyapunov function for a stochastic 
system in terms of the resolvent of the Koopman operator is 
presented in section [TVl The connection between the Lyapunov 
measure and the Lyapunov function is discussed in section [V] 
The set-oriented numerical method for the finite dimensional 
approximation of P-F operator and stability results using the 


finite dimensional approximation are discussed in VI and VII 


respectively. Simulation examples and discussed in Section 


VIII followed by conclusions in section IX 


11. Preliminaries 

The set-up and preliminaries for this section are adopted 
from 0- Consider the discrete-time stochastic dynamical 
system, 

^n+l — ( 1 ) 

where a;„ G X C is a compact set. We denote B{X) as 
the cr-algebra of Borel subsets of X. The random vectors, 
^re assumed independent identically distributed 
(i.i.d) and takes values in W with the following probability 
distribution. 


Prob(^„ £ B) = v{B), Vn, B C W, 


and is the same for all n and v is the probability measure. 
The system mapping T{x,^) is assumed continuous in x and 
for every fixed a; G X, it is measurable in The initial 
condition, xq, and the sequence of random vectors, ^O; Cij • ■ 
are assumed independent. Let W x W = W^. Then, the two 
times composition of the stochastic dynamical system, denoted 
by : X X ^ X, is given by 

Xn+2 = T{T{Xn,^n),^n+l) -■ (Xn, ) , 


where G W^. Since the sequence of random vectors {^n} 
is assumed i.i.d, the probability measure on will simply 
be the product measure, v x v ■.= v^. Similarly, the n-times 
composition of a stochastic system ([T]i, T" : X x IK" —X, 
is denoted by Xn+i = T^{xo,^q), where G PL" with 
probability measure u". 

The basic object of study in our proposed approach to 
stochastic stability is a linear transfer, the Perron-Frobenius 
operator, defined as follows: 

Definition 1 (Perron-Frobenius (P-F) operator): Let 
Af(X) be the space of finite measures on X. The Perron- 
Frobenius operator, P : Af(X) —> Af(X), corresponding to 
the stochastic dynamical system Q is given by 


^tp]{A) = J^XAi.T(^x,y))dv{y)dp{x) 

= Ut-\A)) 


( 2 ) 


for /i G Xi{X), and A G B{X), where xaIx) is an indicator 
function of set A. 

The expectations on f are taken with respect to the probability 
measure, v, and is the inverse image of set A under 

the mapping T{x,f) for a fixed value of i.e.. 


T^\A) = {x:T{x,0&A}. 


Furthermore, if we denote the P-F operator corresponding to 
the dynamical system, : X ^ X, for a fixed value of ^ as 

[Ptj^K^) = J^XA{T{x,^))dp{x) = fi{T~^{A)), 

then, the P-F operator Q can be written as 


[Ftf]{A) = E^[¥t,p]{A). 


The Koopman operator is dual to the P-F operator and defined 
as follows: 

Definition 2 (Koopman operator): Let h G C°(X) be the 
space of continuous function. The Koopman operator, Ut : 
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C°(X) — 7 > C°(X), corresponding to the stochastic system ([^ 
is defined as follows: 

[VTh]{x)= [ hiT{x,y))dv{y)=E^[h{T{x,0)]. 0) 

Jw 

where the expectations are taken with respect to probability 
measure, v. 

Let be a measurable function and y G M(X), define the 
inner product as {h, y) = J^hdy{x). With respect to this 
inner product, the Koopman and P-F operators are dual to 
each other. Using the inner product definition, we can write 
Q as follows: 

(Xv4,PrM) = {^TXA,y) ■ 


d denoted as y d, if y{B) = 0 for all B G B{X) with 
d{B) = 0. 

Definition 8 (Equivalent measure): The two measures, y 
and d, are equivalent [y « d) provided y{B) = 0, if and 
only if d{B) = 0 for S e B{X). 

A. Decomposition of the P-F operator 

Let E = {0}. Hence, E‘^ = X\E.We write T : EUE'^x 
W ^ X. For any set B G B{E'^), we can write 

[FryliB) = Xb(T(x, y))dv(y)dy(x) 

= Ie- fw Xb(T{x, y))dv(y)dy{x). (4) 


Due to the linearity of the scalar product, this implies 
(p„,PTft) = (Urffn, ft), where = X;r=i ctiXAi is the sum 
of a simple function. Since, every measurable function, h, can 
be approximated by a sequence {pn} of simple functions, we 
obtain in the limit, {h,VTd) = {^Th,y). 

Assumption 3: We assume a; = 0 is an equilibrium point of 
system ([^, i.e., T(0,^„) = 0, Vn, for any given sequence 
of random vectors {^n} taking values in set W. 

Assumption 4 (Local Stability): We assume the trivial so¬ 
lution, a: = 0, is locally stochastic, asymptotically stable. In 
particular, we assume there exists a neighborhood O of x = Q, 
such that for all xq G O, 


This is because T{x,^) G B implies x ^ E. Since set E is 
invariant, we define the restriction of the P-F operator on the 
complement set E'^. Thus, we can define the restriction of the 
P-F operator on the measure space Xi(E^) as follows: 

'^iy](B)= [ f XB(T{x,y))dv(y)dy(x), 

JE<^ Jw 

for any set B G B{E'^) and y G M{E‘^). 

Next, the restriction T : E x W ^ E can also be used to 
define a P-F operator denoted by 

[Foy\{B) = f XB(T(x,y))dv{y)dy{x), 

J B 


Prob{r”(a;o,^^) G C>} = 1, Vn > 0, 

and 

Prob{ lim T^{xo,^o) = 0} = 1. 


where y G M.{E) and B C B(E). 

The above considerations suggest a representation of the P- 
F operator, P, in terms of Pq and Pi. Indeed, this is the case, 
if one considers a splitting of the measured space. 


As sumptio n!^ is used in the decomposition of the P-F operator 
in section (II-Ai and Assumption is used in the proof of 
Theorem (lOi. In the following, we will use the notation U{e) 
to denote the e neighborhood of the origin for any positive 
value of e > 0. We have 0 G C/(e) C O. 

We introduce the following definitions for stability of 
stochastic dynamical systems Q- 

Definition 5 (Almost everywhere (a.e.) almost sure stability): 
The equilibrium point, x = 0, is said to be almost everywhere, 
almost sure stable with respect to finite measure, m G M.{X), 
if for every (5(e) > 0 , we have 


m{x G X : Prob{ lim T”(x,Co) 7 ^ 0} > .5} = 0. 

n—foo 

Definition 6 (a.e. stochastic stable with geometric decay): 
For any given e > 0, let U{e) be the e neighborhood of the 
equilibrium point, x = 0. The equilibrium point, x = 0, 
is said to be almost everywhere, almost sure stable with 
geometric decay with respect to finite measure, m G M.[X), 
if there exists 0 < a(e) < 1, 0 < /3 < 1, and K(e) < 00 , 
such that 


M[X)=Mo(BMi, (5) 

where Mo := M(E), Mi := M{E‘^), and 0 stands for 
direct sum. 

Then it follows the splitting defined by Eq. Q, the P-F 
operator has a lower-triangular matrix representation given by 



III. Lyapunov measure and stochastic stability 
W e begin the main results section with the following 
Lemma. 

Lemma 9: Let 

Co = ■ ■ ■ ,?n} G ,lUx...xfU =: fU" 

n 

and F{x,Q) := T’^{x,Q) : X x VF” ^ AT be the notation 
for the n times composition of the map T : X x PF —7 X. 
Then, the Perron-Frobenius operator, Pj? : M{X) -A XI (X), 
corresponding to system mapping F is given by 


m{x G X : Pro 6 {r"(x, Co) G B} > a"} < X/3”, 

for all sets B G B{X \ 17(e)), such that m(B) > 0. 

We introduce the following definition of absolutely contin¬ 
uous and equivalent measures. 

Definition 7 (Absolutely continuous measure): A measure 
y is absolutely continuous with respect to another measure. 


Vf = PtPt ...Ft-. Pt- 

n 


Proof: Let t?(A) = [P'r/r](^)> it then follows from 
Definition for the P-F operator that 


[FTy]{A) = d{A) = E^, 


d{T[,\A)) 


(7) 
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where expectation is taken with respect to probability measure 
V and (A) is the inverse image of set A under the mapping 
T{x,^o) for fixed values of i-C-, 

T-\A) = {xGX :Tix,^o)&A}. 

Hence, 


Following and defining = A, we obtain 







local neighborhood of a; = 0. However, lim„_>.oo [P^m] (B) = 
0 for any set B € B{X \ 17(e)) and, in particular, for B = S. 
Hence, we have m{S) = m{S) = 0. ■ 

Theorem 11: The x = 0 solution for system Q is a.e. 
stochastic stable with geometric decay with respect to finite 
measure, m G Xi{X), if and only if there exists a positive 
constant, -ff(e), and 0 < /3 < 1, such that 

[F^m]{B) < K(3^, Vn>0 

for every set B G B{X \ C/(e)), such that m{B) > 0. 

Proof: We first prove the sufficient part. 


Since are independent with the same probability [P"m](i3) = / / xb{T^ {x, ^Q))dvQ {^Q)dm{x) < 

distribution v, we obtain ^ 


= fw^ fxXB(T(T(x,po),m))dv(yo)dv(yi)dfi(x) 

= [PT2(a;,{i)M](-B)- (8) 

The main results of this lemma then follow from induction. 


Using the lower triangular structure of the P-F operator in 
Eq. (|^, one can write the as follows: 




(9) 


We now state the first main results of the paper on the 
stochastic stability expressed in terms of asymptotic behavior 
of Py. 

Theorem 10: The equilibrium point, a: = 0, for system 0 
is almost everywhere, almost sure stable with respect to finite 
measure, m G Xi{X), if 


lim [¥^m]{B) = 0, 

for every set B G B{X \ 17(e)), such that m{B) > 0. 17(e) is 
the e neighborhood of the equilibrium point, a: = 0, for any 
given e > 0. 

Proof: For any given (5(e) > 0, consider the following 

sets. 


[F'lm]{B) = XB(T"(x,^!}))dvl}(^l^)dm(x) 

= Prob{T'^{x,Q) G B}dm{x). (11) 

Choose 0 < q; < 1, such that - < 1. Let 

Sn = {xGX: Prob{T'^ix,^J}) G B} < a"} 

and 

Sn = {xGX : Pro6{r"(a;,^^) G B} > a^}. 

Then, 

[P!)to](P) = [ Prob{T^{x,^^) G B}dm{x) 

■ISr, 

+ f Prob{T^{x,^^) G B}dm{x). 

Since integrals over both sets are positive, we have 
[¥^m]{B)> [ Prob{T^{x,^l^) G B}dm{x)>a^m{Sn). 
Hence, we have, 

/3” 

m{Sn) < K — , 

Of" 

which is equivalent to 

m{x G X : Prob{T'^{x,^l}) G B} > a"} < iT^”. 


and 


Sn = {x G X : Prob{T'^{x, ^q) G X\U{e)) > 6, for some k > F^r the necessary part, we assume the system is a.e. stochastic 

stable with geometric decay (Definition 0. Construct the 
sets, Sn and Sn, from the sufficiency part of the proof. 
The parameter, a, for this construction now comes from the 
stability definition (Definition 0. We have 


s = n^^,Sn. 


So, set S consists of points with probability larger than (5 to 
end up in set X \ 17(e). Now, since e is arbitrary small and 
from the local stability property of equilibrium point, we have 
0 G U{e) C O. The proof of this theorem follows, if we show 
m(S) = 0. Let S = Sn{X\0). Then, from the property of the 
local neighborhood, m{S) = m{S). Lrom the construction of 
set S, it follows X G S', if and only if Prob{T{x,^) G S) = 1. 
Hence, 

[P'rTO](S) = f Prob{xs{T{x,^)))dm{x) = m{S). (10) 

Jx 

Now, S C S with m(S) = m{S). Since T is continuous in x 
and measurable in we have [Pito](S) = [Pim](S). Using 
(10 1 , we obtain [Pim](S) = m(S). S C X lies outside some 


[P”to](P) = [ Prob{T^{x,^^) G B}dm{x) 

JSr, 

+ [ Prob{T^{x,^^) G B}dm{x). 

JSr, 

Hence, 

[F^m]{B) < a^m{Sn) + to(S„) < a”m(S„) + ST/?". 

Now, since X is assumed compact, we have m{Sn) < M for 
some M < oo. Hence, we have 

[V^m]{B) < 

where K = m.ax{M, K) and P = max(/3, a) <1. ■ 
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Following is the definition of Lyapunov measure introduced 
for stability verification of a stochastic system. 

Definition 12 (Lyapunov measure): A Lyapunov measure, 
jl G Ai(X \ U{e)), is defined as any positive measure finite 
outside the e neighborhood of equilibrium point and satishes 


Allowing 7 = (32, we satisfy the requirements for the Lya¬ 
punov measure. The equivalence of measure, m, the Lyapunov 
measure, p, and the dominance of the Lyapunov measure 
follows from the construction of the Lyapunov measure as 
a inhnite series formula Eq. 0. ■ 


[F,fl]{B) < yp{B) (12) 


for 0 < 7 < 1 and for all sets B G B{X \ U{e)). 

Theorem 13: Consider the stochastic dynamical system 0 
with x = 0 as a locally stable, equilibrium point. Assume 
there exists a Lyapunov measure, p, satisfying Eq. (12i with 
7 < 1. Then, 


1) X = 0 is almost everywhere almost sure stable with 
respect to hnite measure, m, which is absolutely contin¬ 
uous with respect to the Lyapunov measure p. 

2) cc = 0 is almost everywhere stochastic stable with 
geometric decay with respect to measure any finite 
measure, m -< np, for some constant, k > 0. 

Proof: 1) Using the dehnition of Lyapunov measure with 
7 < 1, we obtain 




lim [F'fp\{B) = 0. 


Now, since m ^ p, we have lim„_>oo[IPT’^](-®) = The 
proof then follows by applying the results from Theorem 10 
2) We have 


[F^m]{B) < k[F^p](^B) < K-f^p{B) < K{e)y^, 

where K{e) = Kp{X\U{e)), which, by dehnition of the Lya¬ 
punov measure, is hnite. The proof then follows by applying 
the results from Theorem [TT] ■ 

The following theorem provides for the construction of 
the Lyapunov measure as an inhnite series involving the P- 
E operator. 

Theorem 14: Let the equilibrium point, a; = 0, be almost 
everywhere, stochastic stable with geometric decay with re¬ 
spect to measure, m. Then, there exists a Lyapunov measure 
(Dehnition [T^ p with 7 < 1. Eurthermore, the Lyapunov 
measure is equivalent to measure m (i.e., p ^ m) and the Lya¬ 
punov measure dominates measure m (i.e., p{B) > m{B)). 

Proof: Eollowing the results of Theorem 0 we know 
there exists a positive constant, K, and /3 < 1, such that 

[F'fm]{B) < 

Let P = P 1 P 2 , such that /3i < 1 and P 2 < 1- Hence, we have 


with a = i > 1- Now, construct the Lyapunov measure as 
follows: 


p{B) = (to -I- a[PiTO] -I- -I- ...) {B) 

= Er=o«npH(s). 


(13) 


The above inhnite series is well-dehned and converges because 
a"'[P"TO](i3) < AT/?". Multiplying both sides of (13i with 
/ — aPi, we obtain 


a[F^p\{B)<p{B) 


]_p]{B) < P2p{B). 


IV. Koopman operator and Lyapunov function 
In this section, we describe the connection between the 
Koopman operator and Lyapunov function for a stochastic 
dynamical system. 

Definition 15 (p*^ Moment Stability): The equilibrium so¬ 
lution a; = 0 is said moment exponentially stable with 
p e Z+, if there exist a positive constants, K < 00 , and 
P < 1, such that 

E^j^[\\xn+ir]<KP^\\xor, Vn>0 

for all initial conditions xq G X. 

The results of the following theorem are not new 0^ 0- 
However, the proposed construction of the Lyapunov function 
in terms of the Koopman operator is new. Eurthermore, the 
connection between the Koopman operator and Lyapunov 
function also brings out clearly the dual nature of the Lya¬ 
punov function and the Lyapunov measure for stochastic 
stability verihcation. 

Theorem 16: The equilibrium solution, a; = 0, is 
moment exponentially stable, if and only if there exists a 
nonnegative function V ■. X ^ IR+ satisfying 

a II X ||^< V{x) < 5 II a: ||^, [UrUKa;) < cU(x), 

where a, b, c are positive constants with c < 1. Eurthermore, 

V can be expressed in terms of resolvent of the Koopman 
operator as follows: 

V{x) = {I-VT)-^f{x), 

where f{x) =|| x p. 

Proof: We hrst prove the sufficient part. We have 

A^jlll x„+i ri < [[UtU](x„)] 

< < ^U(xo) < II xo r ■ (14) 

Let, K = - and c = P. Thus, we obtain the desired condition 
for the p*^ moment exponential stability. Eor the necessary 
part, let 

VAr(xo) = 'Lk=oE^^[f{xk+l)] = EfcO^{J-i[pT/](xfe)] 

= Ef=o[Ur/](^o). (15) 

The uniform bound on Vm{x) follows from p*^ moment 
exponential stability. Hence, U(x) = limjv_>.oo Vm{x) is well 
dehned. Eurthermore, 

N 

U(x) = lim V[U^/](xo) = (I - Ut)-V. (16) 

N—>00 ^' 
k=0 

The bounds on the function, V, and the inequality, 
[UtU](x) < cV{x), follow from the construction of the 
function, V. ■ 

The P-E operator and Koopman operator are shown dual to 
each other. In Eqs. 0 and ([T6|, the Lyapunov measure and 
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the Lyapunov function are expressed in terms of an infinite 
series involving the P-F and Koopman operators, respectively. 
Using the duality relationship between the P-F and Koopman 
operators, it then follows the Lyapunov function and the 
Lyapunov measure used for verifying stochastic stability are 
dual to each other. In the following section, we establish 
a precise connection between the two stochastic, stability 
verihcation tools. 


Hence, (following Theorem [T^ , there exists a Lyapunov 
measure satisfying 

dfL{x) — ad[Fi^]{x) = g{x)dm{x). 

Furthermore, the density function corresponding to p{x) sat- 
ishes 


V. Relation between Lyapunov measure and 

FUNCTION 


In this section, we relate Lyapunov measure and Lyapunov 
function. We impose additional assumptions, the system map¬ 
ping 0, i.e., T : X X W ^ X. We assume the system 
mapping T is invertible diffeomorphism with respect to 
state variable, x, for any fixed value of noise parameter, 
For the diffeomorphism, we dehne 


J, ^{x) = 


dT[^ 

dx 


{x) 


where | • | stands for the determinant. 

Lemma 17: Let P-r be the P-F operator for the system 
mapping T ■. X xW ^ X. Then, 

d[P7’TO](a:) = E^[J^^{x)]dm{x). 


ai II X II P< p{x) < 02 II a; II ^ . (17) 

Then, the x = 0 solution is exponentially stable with 
Lyapunov function V obtained as V(x) = p(x)~^. 

2) Let a; = 0 be the moment exponentially sta¬ 

ble with the Lyapunov function V{x) satisfying V{x) < 
/dE^[J^{x)V{T^^(x))] for some (3 < 1. Then, the measure, 

^ (18) 

is a Lyapunov measure satisfying E^[fl{Tf^(B))] < Kfl{B) 
for some k <1 and for all B G Xi{X\U{^) with m{B) > 0. 
7 > 1 is a suitable constant chosen, such that ^ is integrable. 
Proof: 1) 

dpix) — a(i[Pi/i](a;) = g{x)dm{x) 

= V~^{x)dm{x) — aE^[Jf^ {x)V~^ {x))]dm{x){l9) 


If p{x) is the density of an absolutely continuous measure, 
p, with respect to the Lebesgue measure, m, i.e., dp{x) = 
p{x)dm{x), then 

d[¥Tp]ix) = E^[Jf^{x)p{T~^{x))]dm{x). 

Proof: 

[PTm](S)= [ ( XB{Ty{x))dm{x)dv{y) 

Jw Jx 


Hence, we have V ^(x) — aE^lJ^ ^(^))] = 

g{x) > 0. Now, since J^{x) < 1 or Jf^{x) > 1 for all 
values of ^ e W, we have 


1 

nx) 


> aE^ 


1 


Now, using Holder’s inequality, we have 


( 20 ) 


lx 


= [ [ XB{x)dm{Ty ^{x))dv{y). 

Jw Jx 

Xb(x) / Jy^{x)dm{x)dv{y) = / xb{x)E[J 7^{x)\dm{x). 
Jw Jx 


l = Ef 


'v{Ty\x)) 

< Fa 

1 - 

1—1 

1_ 

v{Ty\x))_ 


v{Ty\x))_ 


d^TBn] = E^[J^ ^{x)]dm{x). 


Er 


V{T^^{x)) 


> 


E^[V{Tf^{x))] 


E^[V{T7\x))]. 


( 21 ) 


dp{x) = p{x)dm{x). 

[PT/a](S) = f f XB{Ty{x))dv{y)p{x)dm{x) 

Jx Jw 


Combining ( [SO] ) and pT] ), we obtain 

V{x)<aEi_[V{T-^{x))], a=^<l. (22) 


[ [ XB{x)dv{y)p{Ty^{x))Jy^{x)dm{x). 

Jx Jw 

d^Tp\ = E[p[Tff^ {x))J^^ {x)\dm{x). 


We have the following theorem connecting the Lyapunov 
measure and the Lyapunov function for a stochastic system. 

Theorem 18: Let J^{x) < A < 1 for all ^ G VF and x G X. 
1) Let the equilibrium point, a; = 0, be almost everywhere, 
stochastic stable with geometric decay with respect to the 
Lebesgue measure, m. Assume the Lyapunov measure is 
absolutely continuous with respect to the Lebesgue measure, 
m, with density function p{x), i.e., dp{x) = p{x)dm{x). 


Since V{x) = p ^(a;) is assumed radially unbounded (Eq. 
(17 1 ), then using the results from Theorem and inequality 
(22 1 , we achieve the desired results, the a; = 0 solution is the 
moment exponentially stable. 

2) We have V{x)^ < E^[J^{x)V{T-'^{x))\^. Using the 

Holder inequality, we obtain 

v{xy <mdJj{^)y'^{T-\x))i 

where P = (3^ < 1. 

^_ 

u(x)T Eyjj{x)vyTy\x))] 
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Now, since < A < 1 is uniformly bounded, there exists 
a 7 > 1, sufficiently large, such that 


V{x)^ 


> 




>Ei 


1 

J^ix)V'r{T^-\x)) 


Integrating over set B G B{X \ 17(e)), we obtain 


y 17 '^(x)dm{x) > J ^(a;)17 ^{x))dm{x). 

Using the results from Lemma ( [T7] l and 0, we obtain the 
desired results by letting k = P, i.e., 

< Kfl{B). 


VI. Discretization of the P-F operator 

In this section, we will discuss the set-oriented numerical 
methods proposed for the finite dimensional approximation of 
the P-F operator. For the hnite dimensional approximation of 
the P-F operator, we will make the following assumption on 
the stochastic process. 

Assumption 19: We assume the random vector takes 
hnitely many vector values wi,..., wq with the following 
probabilities 

Prob{^„ = wi} = pe Vn, £ = 


For the hnite dimension approximation of the P-F operator, 
we consider the hnite partition of the state space, X, as 
follows: 




where UjDj = X. The inhnite dimensional measure, p., is 
approximated by ascribing a real number pj to each cell of 
the partition, X. In this way, the inhnite dimensional measure, 
p, is approximated by a hnite dimensional vector, p G K^. 
The hnite dimensional approximation of the P-F operator 
will arise as a Markov matrix of size L x L on the hnite 
dimensional vector space. Using Assumption[T^ the stochastic 
dynamical system, T{x,^) : X xW —i' X, is parameterized 
by hnitely many values that the random variable takes. In 
particular, for each hxed value of a random variable, i.e., 
Cn = Wk, we have T{x, Wk) for k = 1,..., Q. For notational 
convenience, we will also write the mapping for each hxed 
value of random variable as T{x,Wk) =■ Tyj^{x). Hence, 
we have : X -G X for k = 1,... ,Q. With the hnite 
parameterizations in the noise space, the hnite dimensional 
approximation of the P-F operator follows exactly along the 
lines of a deterministic setting. In particular, corresponding to 
a vector, p — {pi, • • • , p^) G K^, dehne a measure on X as 


L 

dp{x) = y^^piKjjx) 

i=l 


dm{x) 

m(Di) 


where m is the Lebesgue measure and Kj denotes the indicator 
function with support on Dj. The approximation, denoted by 


P, is now obtained as 


where 


[Fp]{D,) 



{x){Dj)pi 


dm{x) 

m{Di) 


Q L 






2=1 ^=1 


^ij 


w(T^/(L>j-) n A) 

m{Di) 


(23) 


m the Lebesgue measure. The resulting matrix is non-negative 
and because : Di ^ X for f = 1,..., Q, 


L 

Y,pr/ = h £=i,...,Q, 

7 = 1 

i.e., P is a Markov or a row-stochastic matrix. 

Computationally, several short-term trajectories are used 
to compute the individual entries P^‘- The mapping 
is used to transport M “initial conditions” chosen to be 
uniformly distributed within a set Di. The entry is then 
approximated by the fraction of initial conditions in box, Dj, 
after one iterate of the mapping. In the remainder of the paper, 
the notation of this section is used, whereby P represents 
the hnite-dimensional Markov matrix corresponding to the 
inhnite dimensional P-F operator P. The hnite dimensional 
approximation of the P-F operator can be used to study the 
approximated dynamics of the stochastic dynamical system. In 
particular, the eigenvector with eigenvalue one of the matrix 
P, i.e., 

L 

pP = p, Pi>0, = 1 , 


captures the steady state dynamics of the system. 


VII. Stability in finite dimension 

The hnite dimensional approximation of the P-F operator 
presented in the previous section can be decomposed into 
the lower triangular form similar to the decomposition of the 
inhnite dimensional P-F operator in Eq. •S'- With no loss of 
generality, we assume the equilibrium point at the origin is 
contained inside the cell, Di, and the partition is sufficiently 
hne so Di C O, where O is the local neighborhood of 
the stable equilibrium point. We decompose X into two 
complimentary partition as follows, 

T’o = {Z?i}, X^={D2,...,Dl}, (24) 

with domains Xq = Di and Xi = Since the local 

stable equilibrium point at the origin is contained in the cell, 
Di, there exists a left eigenvector with eigenvalue one to 
the Markov matrix, P, of the form pQ = (1,0..., 0), i.e., 
do — BoP- The existence of the eigenvector can be used 
to decompose the Markov matrix, P, in the following upper 
triangular structure. Let M = R^,Mq = K, and Mi = 
denote the hnite dimensional measure space associated with 
the partition X,Xo, and Xi, respectively and as dehned in 










Then, for the splitting M = Mq ©Mi, the P matrix has where m is the Lebesgue measure. We need to show Pi < 1. 
the following lower triangular representation. Denote, 


P = 




where Pq = 1 and maps Pq : Mq — )■ Mq and Pi : Mi — )■ Mi 
is the sub-Markov matrix with row sum less than or equal to 
one. We refer the readers to for details on the decompo¬ 
sition in finite dimension. Our goal is to study stability with 
respect to initial condition starting from partition Xi using the 
sub-Markov matrix. Pi. In a discrete finite dimensional setting, 
stability is expressed in terms of the transient property of the 
stochastic matrix Pi. 

Definition 20 (Transient states): A sub-Markov matrix Pi 
has only transient states, if P" —)■ 0, element-wise, as n —t oo. 
Transience of Pi is shown to imply a weaker notion of stability 
referred to as coarse stochastic stability and defined as follows. 


Definition 21 (Stochastic Coarse Stability): Consider the 
finite partition, Xi, of the complement set, Xi = X \ Xq. 
The equilibrium point, x = 0, is said to be stochastic coarse 
stable with respect to the initial condition in Xi, if for an 
attractor set, B C U C Xi there exists no subpartition 
S — {Psi 5 • ■ •, Dsi} in Xi with domain S = such 

that B C S CU and for all x S S' Prob{T{x, G S} = 1 . 

For typical partitions, coarse stability means stability mod¬ 
ulo attractor sets, B, with domain of attraction, U, smaller 
than the size of cells within the partition. In the infinite¬ 
dimensional limit, where the cell size (measure) goes to zero, 
one obtains stability modulo attractor sets with measure 0 
domain of attraction. 

Theorem 22: Let the equilibrium point, x = 0 G Xq C X, 
with local domain of attraction contained inside Xq, Pi is the 
sub-Markov operator on AI(P°). Pi be its finite dimensional 
approximation obtained with respect to the partition Xi of the 
complement set Xi — X\Xq. For this, we have the following 

1) Suppose a Lyapunov measure, fi, exists such that 

[Pi/2](P) </2(P) (25) 

for all B C B{Xi), and additionally, fi = m, the 
Lebesgue measure. Then, the finite-dimensional approx¬ 
imation, Pi, is transient. 

2) Suppose Pi is transient, then the equilibrium point, x = 
0, is coarse stable with respect to the initial conditions 
in Xi. 

Proof: Before stating the proof, we claim for any two 
sets. Si and S, such that ^i C S', if /r « m, then 

n{Si) = fJ-(S) m.(Si) = m(S). (26) 


Denote Si := S\Si as the complement set. We have, /i(Si) = 
p(S) implies p.(Si) = 0 which, in turn, implies m(S1) = 0 
and, thus, m(Si) = m{S). 

1. We first present a proof for the simplest case where the 
partition, Xi, consists of precisely one cell, i.e., Xi = {Pl}. 
In this case. Pi G [0,1] is a scalar given by 


Q 

Pi = '^Pk 


k^l 


miT^^iPL) n Dl) 
m(DL) 


Sfc = {xGPL: T^,(x)gPl}, k = l,...,Q. (28) 


Clearly, Sk C Dl and the existence of Lyapunov measure p. 
satisfying Eq. p5| ) implies 

p(Sfe) = [P^''p](Pl) < ftiPh), k = l,...,Q. 


Now, 


Q 

E 


Pkp{Sk) = '^Pk[Vi’‘ iA{Dl)] < p{Dl)- 


k=l 


Using (26 1 , m(Sk) m(DL) and since Sk C Dlj we have 
m{Sk) < m(DL) for fc = 1,..., <5- Hence, 

m(Sk) 

Pi— / Pk - . ^ N < 1 




<Dl) 


is transient. 

We prove the result for the general case, where Xi is a finite 
partition, by contradiction. Suppose Pi is not transient. Then, 
using the general result from the theory of finite Markov chains 
1531, there exists at least one non-negative invariant probability 

vPi=v. (29) 


vector, V, such that 


Let, 

S = {x & Di : Vi> 0}, Si = {x € S : Tyj^(x) G S}, 
for £ = 1,... ,Q. It is claimed 

m{Si) = m(S), £ = 1,...,Q. (30) 

We first assume the claim to be true and show the desired 


contradiction. Clearly, Si C S and if the claim were true, (261 
shows 

fl{Se) = fl{S), e = l,...,Q. (31) 

Next, because S C Xi, 

^iKS) =EtiPiPiTf;!is)nXi) 

> T,t=iPipiTwH^) ns) = J2f^iPifi{Si). 
This, together with Eq. ( [3T] i, gives 

ViHS) > fi{s) 

for a set S with positive Lebesgue measure. This contradicts 
Eq. (251 and proves the theorem. 

It remains to show the claim. Let {ik}k=i be the indices 
with > 0. Equation . (29i gives 


Q 




= for r = 1,... ,1. 


k=l 


Taking a summation, X]r=i’ either side gives 


k — l r—1 £—1 


,W£-\ _ -I 

^kjr — 


(27) 
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Since individual entries are non-negative and z/ is a probability 
vector, this implies 


l Q 

r=l £=1 


iikj, 


= 1 k = 1,... ,1. 


Using formula (231 for the individual matrix entries, this gives 

l Q 

n AJ = rn{Di^), 

r=l ^=1 


Q I 

AJ = to(AJ. 

£=1 r=l 

Since Ar disjoint for any fixed W£, 

\Jr{x ■■ T^iix) G Dj^} = {a; : A^(a:) G UrDj^}, 

i.e., ur“/(Ar) = T~l{UDj^). Furthermore, since is 
one-to-one, we have (T-/(A,.J n AJ n (r-/(Av.J n 
AJ = 0 as 


{a: G Afc : T^eix) G Dj^^}r){x G Di^ : A^(a;) G Ar^} = 0 

for ri ^ r 2 . Hence, EtiAJ ^ AJ = 
m{Di^). Therefore, 

Q 

^P£m(T-/(uf^iAJ n AJ = to(AJ- 

However, by construction, S = and, thus. 


^P£m(A/(S') n AJ = m(Aj- 




Taking a summation on both sides J2k=v obtain 
Q Q 

'^pem{Se) = m{S) = '^pem{S). (32) 


£=1 


Since S£ C S, we have m^Si) < m{S). From (32i, we obtain 
Q 

'^P£{m{Se) - m{S)) = 0. 


Since > 0 and m{Si) < m{S), we conclude m{S£) = 
m{S). 

2. Suppose Pi is transient. To show the equilibrium point 
is coarse stable, we proceed by contradiction. Using defini¬ 
tion 21 if the equilibrium point is not coarse stable, then 


there exists an attractor set i? C C/ C with a sub-partition 
S = {Ds ^,..., Dsi}, S = uAi-Dsfc such that B C S C U and 
Prob{T{x,^) G = 1 for all x G S'. 


Prob{T{x,0 G S} = 1 = ^PkXs{TwAx)) 

£=1 

for X G S. This implies S is left invariant by each for 
£ = 1,... ,Q. Hence, 

^ »^('C/(-Pa)n AJ ^ 
m(Aj 


whenever Dj ^ S. Since, A? : S —)■ S, 

Q i 

£=\ j=l 

i.e.. Pi is a Markov matrix with respect to the finite partition, 
S. From the general theory of the Markov matrix | [53j , there 
exists an invariant probability vector, i/, such that ly ■ Pi = v 
for all n > 0 and Pi is not transient. ■ 

There are various different equivalent ways of computing 
the finite dimensional approximation of Lyapunov measure 
using the finite dimensional approximation of the P-F operator. 
In particular, if Pi is transient, then the Lyapunov measure p 
can be computed using the following formulas. 

1) Compute the Lyapunov measure using an infinite series 
formula, 

jl = m ■ {I — Pi)~^ = m + m-Pi+m-Pi+..., 

for row vector m > 0 (element wise). For example m 
could be taken as row vector of all ones corresponding 
to the finite dimensional approximation of Lebesgue 
measure. 

2) Compute the Lyapunov measure as a solution of the 
linear program 


p{al — Pi) > 0, a <1. 

VIII. Examples and Simulation 

Example 23: We consider an inverted pendulum example 
with a stochastic damping parameter, 


X’l = X 2 - 

X 2 = — sinxi — (^-f 0.7 )x2. (33) 

For the purpose of simulation, the continuous system is 
discretized with the time step of discretization. At — 0.1. 
The phase space, X, for simulation is taken to be equal to 
X = [— 7r,7r] X [— 7r,7r]. For the purpose of compacting the 
state space, we identify —tt and tt along the x and y axis. 
The damping parameter,, is assumed stochastic with zero 
mean and uniform distribution; hence, of the form [—a, a]. In 
the following, we present simulation results by changing the 
variance, i.e., a, of the random variable, while keeping the 
mean zero and verifying the stochastic stability of the origin. 

In Figs. [T] an d|g we show the plot for the Lyapunov measure 
with random variable, supported on interval [—0.5,0.5] 
and [—0.75,0.75], respectively. The existence of Lyapunov 
measure implies the origin is almost everywhere almost sure 
stable. When support of the random parameter, is increased 
to [—1,1], the system fails to have Lyapunov measure and 
is confirmed from the invariant measure plot shown in Fig. 
1^ From Fig. we notice the invariant measure has support 
around the origin. This is in contrast to the invariant measure 
plots when ^ is supported on [—0.5, 0.5] and [—0.75, 0.75] and 
shown in Figs. [T]and[^ The invariant measure in these cases 
is supported at the origin and is denoted by a blue dot at the 
origin in Figs. and 
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xIO"^ 



Xl 

Fig. 1. Lyapunov measure plot with support of ^ E [—0.5, 0.5]. 


xIO'^ 



Xl 

Fig. 2. Lyapunov measure plot with support of ^ £ [—0.75, 0.75]. 



Xl 

Fig. 3. Invariant measure plot with support of ^ G [—1,1]. 


Example 24: The second example is the stochastic coun¬ 
terpart of the deterministic almost everywhere stable system 
example from p^ . 

X = —2a; + x^ —j/^, 

V = -61/(1 + 0 + 2a;t/, (34) 

where ^ is a random variable with zero mean and uniformly 


distributed between [—a, a]. The deterministic system has four 
equilibrium points (0,0), (2,0) and (3,±-\/3) of which the 
origin is stable equilibrium and the remaining are unstable. For 
the stochastic system, two of the equilibrium points - one at 
the origin and equilibrium point at (2, 0) are preserved. For the 
purpose of simulation, the system is discretized in time with 
the time step for discretization chosen to be equal to Af = 0.1. 
The state space X is taken to be A = [—4,4] x [—4,4]. For 
the purpose of compactness, —4 and 4 are identified along the 
X— axis. 

For better visualization, we plot the logarithm of the Lya¬ 
punov measure plot for varying variance of the random vari¬ 
able, by changing a, while retaining its zero mean. In Figs, 
j^and]^ we show the log plot for the Lyapunov measure with ^ 
supported between [—0.25, 0.25] and [—0.5, 0.5], respectively. 
The existence of a Lyapunov measure in these plots implies 
the origin is almost everywhere almost sure stable for these 
statistics of random variable. When the support of the random 
variable is changed to [—0.1,0.!], the origin is no longer 
almost everywhere almost sure stable. In Fig. [^ we show the 
plot for the invariant measure supported in the neighborhood 
of the origin. This is in contrast to the invariant measure plot 
for the cases when ^ S [0.25, 0.25] and ^ S [0.5,0.5]. For both 
cases, the invariant measure is supported only at the single cell 
containing the origin. 



Fig. 4. Log plot of Lyapunov measure with noise support ^ G [—0.25, 0.25]. 


IX. Conclusions 

Weaker set-theoretic notion of almost everywhere stability 
for stochastic dynamical system is introduced. Linear P-F 
operator-based Lyapunov measure as a new tool for stochas¬ 
tic stability verification is introduced. Duality between the 
Lyapunov function and the Lyapunov measure is established 
in a stochastic setting. Set-oriented numerical methods are 
proposed for the finite dimensional approximation of the 
Lyapunov measure. The finite dimensional approximation in¬ 
troduce further weaker notion of stability refereed to as coarse 
stochastic stability. The framework developed in this paper 
can be easily extended to the case where the noise, ,f„, forms 
a Markov processes. This can be done by employing the P- 
F operator defined for systems with Markov noise processes 

izi- 
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-2 0 2 
xi 


Fig. 5. Log plot of Lyapunov measure with noise support ^ E [—0.5, 0.5]. 



-2 0 2 
Xi 


Fig. 6. Invariant measure plot for ^ E [—1,1]. 
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